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Abstract 


Campaign  Assessment  in  Counterinsurgency:  Reinventing  the  Wheel,  by  MAJ  James  C.  Bithom, 
United  State  Army,  40  pages. 

Strategic  and  political  leaders  have  determined  that  the  US  and  1SAF  campaign  assessments  were 
neither  useful  nor  accurate.  The  DOD  1230  biannual  report  titled  the  Report  on  Progress  toward 
Security  and  Stability  in  Afghanistan  provides  comprehensive  information  to  strategic  and 
political  leaders  regarding  the  progress  of  the  United  States’  strategy  in  Afghanistan.  The  report, 
however,  has  largely  provided  indicators  of  progress  based  off  a  single  variable — enemy  initiated 
ambushes.  As  attention  drew  to  the  failure  of  1SAF  to  provide  accurate  reporting,  a  degree  of 
mistrust  grew  between  strategic  and  political  leaders  and  the  military  leaders  in  Afghanistan.  In 
turn,  strategic  leaders  made  only  partially  informed  decisions  that  did  not  reflect  accurately  the 
progress  on  the  ground.  The  July  2013  DOD  1230  report  touted  a  change  in  metrics,  placing  a 
greater  emphasis  on  variables  such  as  security  force  development  and  economic  growth.  The 
announcement  of  new  metrics  raised  an  obvious  question,  were  the  new  measures  of  performance 
in  Afghanistan  actually  better,  and  why  had  change  taken  so  long? 

Answering  this  research  question  required  four  steps.  First,  it  was  necessary  to  determine  the 
importance  of  assessments  and  measures  of  performance  in  the  development  of  strategy  in  the 
COIN  environment.  Second,  it  was  necessary  to  examine  how  1SAF  initially  developed  its 
assessment  plan.  Next,  it  was  necessary  to  examine  a  historical  instance  of  campaign  assessments 
in  COIN.  The  US  and  MACV  in  Vietnam  experienced  similar  complications  throughout  the 
Vietnam  War.  Initially  accused  of  intentionally  bloating  casualty  figures,  MACV  made  a 
concentrated  effort  to  develop  a  more  accurate  assessment  model  to  inform  progress.  The 
improved  model,  the  Flamlet  Evaluation  System,  analyzed  a  host  of  variables  across  South 
Vietnam  to  assess  progress  over  time.  Lastly,  by  comparing  the  efforts  in  Vietnam  with  those  in 
Afghanistan  it  was  possible  to  identify  assessment  concepts  that  would  have  been  applicable  in 
Afghanistan. 

The  evidence  ultimately  shows  that  as  the  US  and  1SAF  sought  to  develop  assessment  measures, 
they  replicated  the  errors  made  in  Vietnam.  Additionally,  despite  advances  in  technology,  ISAF’s 
assessment  plan  did  not  reach  the  level  of  sophistication  that  MACV  achieved  in  Vietnam. 
Although  the  HES  was  criticized  during  the  Vietnam  War,  the  iterative  attempts  by  MACV  to 
improve  its  assessment  plan  provided  useful  lessons  for  the  US  and  ISAF  in  Afghanistan. 
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Introduction 


Going  forward,  we  will  not  blindly  stay  the  course.  Instead,  we  will  set  clear 
metrics  to  measure  progress  and  hold  ourselves  accountable.  We’ll  consistently  assess 
our  efforts  to  train  Afghan  security  forces  and  our  progress  in  combating  insurgents.  We 
will  measure  the  growth  of  Afghanistan’s  economy,  and  its  illicit  narcotics  production. 
And  we  will  review  whether  we  are  using  the  right  tools  and  tactics  to  make  progress 
towards  accomplishing  our  goals. 

—  President  Barack  Obama,  On  a  New  Strategy >  for  Afghanistan  and  Pakistan 
The  Department  of  Defense  (DOD)  1230  Report,  Report  on  Progress  toward  Security 
and  Stability  in  Afghanistan  provides  comprehensive  information  to  strategic  and  political  leaders 
regarding  the  progress  of  the  United  States’  strategy  in  Afghanistan.1  Since  2008,  every  six 
months  the  DOD  prepares  and  submits  the  report  to  the  Senate  and  the  House  of  Representatives. 
When  preparing  this  report,  the  DOD  receives  input  from  the  Secretary  of  State,  the  Director  of 
the  Office  of  Management  and  Budget,  the  Director  of  National  Intelligence,  the  Administrator  of 
the  Drug  Enforcement  Administration,  and  the  Administrator  of  the  United  States  Agency  for 
International  Development.2  The  July  2013  edition  of  the  DOD  1230  Report  included  a  section 
titled,  “A  Note  on  Metrics.”  The  note  commented  on  the  overreliance  on  Enemy  Initiated 
Ambushes  (EIAs)  as  the  sole  measure  of  progress,  which  marked  DOD’s  recognition  that  the 
reliance  on  that  one  measure  to  assess  progress  was  not  useful.3  “As  the  conflict  has  evolved,  it 
has  become  clear  that  a  tally  of  EIAs  is  not  now,  nor  was  it  ever,  the  most  complete  measure  of 
the  campaign’s  progress.”4  This  DOD  revelation  occurred  twelve  years  into  US  and  International 
Security  Assistance  Force-Afghanistan  (ISAF-A)  operations  in  Afghanistan.  To  rectify  this 


1  Department  of  Defense,  DOD  1230:  Report  on  Progress  toward  Security  and  Stability 
in  Afghanistan  Report  to  Congress  (Washington,  DC:  2012),  1. 

2  Ibid. 

3  Ibid.,  8. 

4  Ibid. 
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problem,  the  July  2013  report  stated  that  future  reports  would  place  a  greater  emphasis  on  the 
measurement  of  Afghan  National  Security  Forces  (ANSF)  development  along  with  economic 
development  and  improvements  in  governance.5  Flowever,  despite  DOD’s  delay  in  providing 
useful  measures,  others  recognized  the  need  before  2013.  Military  leaders  and  strategic 
commentators  noted  DOD’s  failure  to  report  on  more  holistic  performance  measures  years  earlier. 
Unlike  the  US  in  Iraq  and  largely  due  to  requests  made  by  media  outlets  and  political 
commentators,  senior  military  officers  did  not  classify  all  the  assessment  material,  which  in  turn 
has  made  this  research  possible.6  However,  it  is  reasonable  to  conclude  that  classified  information 
related  to  assessments  exists,  but  will  be  unavailable  for  some  time.  As  such,  this  research  is 
limited  to  the  released,  unclassified  data  on  assessments  in  Afghanistan. 

Upon  assuming  command  of  1SAF  in  2009,  General  Stanley  McChrystal  published  his 
“Commander’s  Initial  Assessment,”  which  reported  on  the  state  of  operations  in  Afghanistan.7 
Within  Section  V  of  the  assessment,  McChrystal  pointed  to  the  absence  of  useful  metrics  and  that 
“. .  .  [1SAF]  must  develop  effective  assessment  architectures  ...  to  measure  the  effects  of  the 
strategy,  assess  progress  toward  key  objectives,  and  make  necessary  adjustments.”8  In  2010,  the 
senior  intelligence  officer  in  Afghanistan,  Lieutenant  General  Michael  Flynn,  assessed  the  US 
intelligence  community’s  performance  in  Afghanistan.  His  report,  Fixing  Intel :  A  Blueprint  for 
Making  Intelligence  Relevant  in  Afghanistan  painted  a  bleak  picture  of  the  US  intelligence 
community  in  Afghanistan.  The  report  stated  that  intelligence  organizations  were  overly  focused 

5  Department  of  Defense,  DOD  1230:  Report  on  Progress  toward  Security  and  Stability 
in  Afghanistan  (July  2013),  8. 

6  Ben  Connable,  Embracing  the  Fog  of  War:  Assessment  and  Metrics  in 
Counterinsurgency  (Washington,  DC:  National  Defense  Research  Institute,  2012),  206. 

7  William  J.  Gregor,  “Military  Planning  Systems  and  Stability  Operations,”  PRISM 
Magazine,  no.  3  (June  2010),  100. 

8  Ibid. 
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on  the  threat,  isolated  from  organizations  outside  of  the  intelligence  community,  and  not 
structured  distill  to  relevant  information  from  the  complex  Afghan  environment.9  Also  in  2010, 
well-known  sociologist  Amitai  Etzioni  reviewed  General  David  Petraeus’  five  metrics  employed 
in  Afghanistan,  formally  known  as  measures  of  effectiveness  (MOE)  or  measures  of  performance 
(MOP).  In  his  September  2010  article,  “Beware  of  Generals  Carrying  Metrics,”  Etzioni 
questioned  the  correlation  between  the  metrics  employed  and  the  prescribed  strategic  and 
operational  objectives  in  Afghanistan.  He  further  observed  that  the  measures  used  strongly 
likened  themselves  to  the  body  counts  used  in  Vietnam  prior  to  the  establishment  of  the  Hamlet 
Evaluation  System  (HES). 10  Etzioni  observed  that  security  metrics  used  at  the  time,  primarily 
enemy-initiated  ambushes,  were  analogous  to  those  measures  used  early  on  in  Vietnam. 11  Finally, 
a  2013  Center  for  Strategic  and  International  Studies  (CS1S)  report,  The  Afghan  War  in  2013, 
Meeting  the  Challenges  of  Transition  stated,  “1SAF  has  reported  on  carefully  selected  indicators 
of  tactical  progress-the  kinetics-in  a  war  that  is  actually  a  war  of  political  attrition.”12  The 
amount  of  attention  paid  to  ISAF’s  performance  measures  by  political  commentators  and 
organizations  indicates  that  many  do  not  find  the  metrics  useful  and  they  doubt  the  accuracy  of 
the  reports. 


9  Michael  T.  Flynn,  Matthew  Pottinger,  and  Paul  D.  Batchelor,  Fixing  Intel:  A  Blueprint 
for  Making  Intelligence  Relevant  in  Afghanistan  (Washington,  DC:  New  American  Century, 
2010),  2-3. 

10  Amitai  Etzioni,  “Beware  of  Generals  Carrying  Metrics,”  Huffington  Post,  September 
30,  2010,  accessed  October  12,  2014,  http://www.huffmgtonpost.com/amitai-etzioni/beware-of- 
generals-carryi_b_745343  .html. 

11  Anthony  H.  Cordesman,  Bryan  Gold,  and  Ashley  Hess,  The  Afghan  War  in  2013: 
Meeting  the  Challenges  of  Transition,  Vol.  1  (Washington,  DC:  Center  for  Strategic  and 
International  Studies,  2013),  14. 

12  Ibid. 
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The  purpose  of  campaign  assessments  is  to  inform  strategic  and  political  leaders.  As 
such,  policy  makers  and  operational  and  tactical  commanders  require  measures  that  clearly 
indicate  that  the  actions  taken  are  succeeding. 13  Beginning  in  2013, 1SAF  claimed  it  had 
developed  a  new  approach  to  security  metrics  drawn  from  a  balance  of  several  environmental 
factors.  The  implementation  of  a  new  approach  leads  to  an  obvious  question  “Has  the 
development  of  new  measures  of  performance  in  Afghanistan  actually  improved  assessments?” 

Answering  this  research  question  required  four  steps.  First,  it  was  necessary  to  determine 
the  importance  of  assessments  and  measures  of  performance  in  the  development  of  strategy  in  the 
Counterinsurgency  (COIN)  environment.  Determining  the  importance  of  assessments  revealed 
what  performance  measures  and  assessments  ought  to  do,  and  what  information  they  provide 
decision  makers.  Additionally,  investigating  the  use  of  assessments  in  strategy  development  made 
clear  how  the  strategy  was  undermined  when  due  diligence  was  not  applied  to  the  development  of 
performance  measures. 

Second,  it  was  necessary  to  examine  how  1SAF  initially  developed  its  assessment  plan. 
This  step  included  a  review  of  the  resources  that  were  available  to  guide  the  development  of 
performance  measures  and  their  usefulness.  Because  reports,  such  as  the  DOD  1230,  inform 
strategic  leaders,  it  was  also  essential  to  understand  their  reactions  to  the  initial  measures,  and  to 
determine  whether  they  considered  the  measures  useful  for  decision-making. 

Third,  Etzioni  suggested  there  were  valuable  lessons  available  by  investigating  the 
Vietnam  War  strategy  assessments.  History  revealed  that  the  pacification  effort  in  Vietnam 
experimented  with  a  variety  of  metrics  to  assess  campaign  progress.  Analysts  lambasted  the  use 
of  enemy  body  counts  to  measure  effectiveness.  That  critique  ultimately  led  to  the  creation  of  the 


13  Connable,  Embracing  the  Fog  of  War:  Assessment  and  Metrics  in  Counterinsurgency, 
52. 
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Hamlet  Evaluation  System  (HES),  a  system  that  focused  on  guerilla  activity,  local  economic 
freedom,  and  attitudinal  assessments  within  the  various  South  Vietnamese  hamlets.14  The  efforts 
by  MACV  in  Vietnam  provide  a  useful  basis  for  assessing  the  development  of  performance 
measures  by  1SAF  in  Afghanistan. 

Fourth,  by  comparing  the  efforts  in  Vietnam  with  those  in  Afghanistan  it  was  possible  to 
identify  assessment  concepts  that  would  have  been  applicable  in  Afghanistan.  The  evidence 
ultimately  shows  that  as  the  US  and  1SAF  sought  to  develop  assessment  measures,  they  replicated 
the  errors  made  in  Vietnam.  Additionally,  despite  advances  in  technology,  ISAF’s  assessment 
plan  did  not  reach  the  level  of  sophistication  that  MACV  achieved  in  Vietnam.  Although  the  HES 
was  criticized  during  the  Vietnam  War,  the  iterative  attempts  by  MACV  to  improve  its 
assessment  plan  provided  useful  lessons  for  the  US  and  ISAF  in  Afghanistan. 

Campaign  Assessment  and  Measures 

A  campaign  assessment  serves  two  purposes.  One,  it  informs  policymakers  and,  two  it 
shapes  decisions  made  by  operational  headquarters.  To  meet  the  two  assessment  goals  often 
requires  two  separate  reports — one  tailored  for  policy  makers,  “up  or  out,”  and  another  for 
operational  units,  “down  and  in.”15  Regardless  of  the  intended  audience,  the  assessment  should 
inform  decisions  on  resource  allocation,  initiatives,  and  changes  to  strategy.  A  useful  assessment 
helps  a  military  organization  to  understand  how  well  it  has  performed  its  mission,  both  outputs 
and  achievements.16  Several  military  publications  describe  and  define  assessments.  The  United 


14  Historical  Branch,  Military  Assistance  Command,  Vietnam,  In  Command  History, 
United  States  Military  Assistance  Command,  Vietnam,  1969  Vol.  1  (Washington,  DC: 
Headquarters,  Department  of  the  Army,  1975),  13. 

15  Connable,  Embracing  the  Fog  of  War:  Assessment  and  Metrics  in  Counterinsurgency, 
32. 

16  Boby  Claflin,  Dave  Sanders,  and  Greg  Boylan,  Improving  Analytical  Support  to  the 
Warfighter:  Campaign  Assessments,  Operational  Analysis,  and  Data  Management  (Alexandria, 
VA:  Military  Operations  Research  Society,  2010),  15. 
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States  Army  Commander’s  Appreciation  and  Campaign  Design  (CACD)  describes  assessment  as 
an  opportunity  to  learn,  and  lists  the  five  outcomes  from  a  well-informed  assessment.  The 
publication  further  asserts  that  the  Army  has  failed  to  put  sufficient  effort  into  achieving  the  final 
benefits  of  information  on. 

•  How  to  execute  the  planned  course  of  action  for  a  specific  operation; 

•  Whether  another  course  of  action  needs  to  be  adopted; 

•  Whether  the  operational  design  based  on  the  problem  frame  is  producing  results; 

•  Whether  the  problem  framing  needs  adjusting;  and  whether  the  learning  mechanisms 
of  the  organization  are  tuned  to  the  particular  operational  problem. 17 

The  development  of  useful  assessments  in  COIN  has  proven  an  elusive  goal.  One  of  the 
more  difficult  aspects  of  assessment  is  the  selection  of  measures  and  success  criteria  to  inform  the 
overall  assessment.  As  the  CACD  has  implied,  there  is  a  dearth  of  military  literature  on  the 
selection  of  performance  measures.  In  the  absence  of  military  doctrine,  complexity  theory 
provides  a  substitute  with  which  understand  the  importance  of  performance  measures. 

According  to  Robert  Axelrod’s  Harnessing  Complexity,  the  logic  behind  success  criteria 
is  simple.  If  an  agent  cannot  determine  whether  a  strategy  is  succeeding  or  failing,  then  how  does 
an  agent  determine  what  should  be  done?  Assessment  in  complex  environments  is  not  easy,  and 
often  leads  to  the  identification  of  more  than  one  criterion  to  assess  results.  Additionally,  the 
measures  may  require  adjustment  over  time  as  agents  react  to  the  measures.18 

Attributing  success  or  failure  incorrectly  easily  leads  to  a  misconstrued  strategy.  The 
variables  used  as  strategy  performance  measures  are  critical  for  making  a  useful  assessment  and 
ultimately  for  making  correct  decisions.  Although  measures  may  require  adjustment  over  time,  it 
is  also  important  to  understand  that  when  performance  measures  are  radically  changed  it  becomes 


17  Headquarters,  Department  of  the  Army,  TRADOC  Pamphlet  525-5-500:  Commander’s 
Appreciation  and  Campaign  Design  (Washington,  DC:  Government  Printing  Office,  2008),  18. 

18  Robert  Axelrod  and  Michael  D.  Cohen,  Harnessing  Complexity:  Organizational 
Implications  of  a  Scientific  Frontier  (New  York:  Simeon  &  Schuster,  2000),  124-125. 
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more  difficult  to  observe  long-term. 19  This  highlights  the  importance  of  longitudinal  data,  or 
performance  measures  that  reflect  a  strategy’s  success  or  failure  over  significant  periods.20 
Consequently,  it  is  important  to  define  consistent  variables  and  performance  measures  at  the  start 
of  a  campaign  rather  than  in  the  midst  of  operations. 

To  understand  the  value  of  assessments  to  a  campaign  strategy,  it  is  important  to 
understand  the  impact  of  performance  measures  on  assessments.  The  selection  of  performance 
measures  requires  deliberate  and  careful  consideration  as  they  inform  an  assessment  by  serving  as 
a  conduit  of  information  for  decision-makers.  The  performance  measures  developed  by  the  US 
and  1SAF  in  Afghanistan  were  solely  focused  on  lethal  events  and  did  not  accurately  inform 
campaign  progress.  Several  factors  such  as  a  lack  of  guiding  doctrine,  unclear  strategic  guidance, 
and  a  convoluted  headquarters  contributed  to  the  problem.  The  following  section  will  explore 
how  the  US  and  1SAF  developed  a  campaign  assessment  plan  and  provide  context  for  the  failure 
to  develop  useful  performance  measures. 

Uncertainty  in  Progress 


Afghanistan  and  Irrelevant  Metrics 

Given  the  importance  of  assessment  criteria  for  determining  progress  during  a 
counterinsurgency  operation,  the  lack  of  effort  to  assess  progress  in  Afghanistan  until  2008  seems 
strange.21  Following  relative  success  against  Taliban  forces  in  the  Hindu  Kush  Mountains  of 
Afghanistan  in  2002,  the  US  failed  to  define  the  next  or  final  objective.  President  George  W. 

Bush  provided  a  vision  of  the  future  in  a  speech  at  the  Virginia  Military  Institute.  Bush  said,  “We 


19  Jeffrey  Reilly,  Operational  Design:  Distilling  Clarity  from  Complexity  for  Decisive 
Action  (Maxwell  Air  Force  Base:  Air  University  Press,  2012),  1 14-115. 

20  Ibid.,  115. 

21  Ibid.,  207. 
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will  stay  until  the  mission  is  done.  We  know  that  true  peace  will  only  be  achieved  when  we  give 
the  Afghan  people  the  means  to  achieve  their  own  aspirations.”22  The  statement  left  the  US 
mission  in  Afghanistan  ambiguous  and  failed  to  define  the  enemy  precisely.  When  the  priority  of 
effort  shifted  to  Iraq  in  2003  Afghanistan  became  an  afterthought  in  the  minds  of  strategic  policy 
makers.  However,  by  2008  a  resurgent  Taliban  and  hosts  of  other  insurgent  groups  re-infiltrated 
from  save  havens  in  Pakistan  and  resumed  the  offensive.  In  the  summer  of  2008,  hostile  acts 
spiked  up  to  four  hundred  per  week,  and  prompted  senior  leaders  to  pay  attention.23  With  Iraq 
finally  winding  down,  1SAF  commander  General  David  McKieman  made  a  request  for  an 
additional  30,000  troops.  Eventually,  and  then  only  under  the  command  of  another  1SAF 
commander,  Obama  partially  filled  that  request.  As  it  became  clear  that  Afghanistan  was  to 
become  a  greater  priority,  serious  efforts  began  to  improve  campaign  planning  and  assessment. 
Counterinsurgency  became  the  expected  approach  in  Afghanistan  once  Obama  appointed  General 
Stanley  McChrystal  to  lead  1SAF  in  2009.  The  hope  was  to  replicate  the  type  of  success  seen  in 
Iraq,  an  outcome  publically  attributed  to  David  Petraeus.24 

Making  a  comparison  between  metrics  used  to  assess  success  in  Vietnam  and 
Afghanistan  is  not  easy.  Both  wars  spanned  more  than  a  decade  and  the  measures  changed. 
Reviewing  the  early  years  in  Afghanistan  was  not  productive  because  an  assessment  plan  was  not 
in  place.  However,  in  2008,  when  the  surge  in  Afghanistan  was  in  the  offering  there  was  a  serious 
concern  for  sound  assessment  and  useful  metric.  Thus,  any  evaluation  of  the  development  of 


22  Daniel  Bolger,  Why  We  Lost:  A  General ’s  Inside  Account  of  the  Iraq  and  Afghanistan 
Wars  (New  York:  Harcourt  Publishing  Company,  2014),  136. 

23  Ibid.,  348. 

24  Ibid.,  352. 
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assessment  measures  needed  to  start  at  2008  and  begin  by  examining  the  doctrinal  guidance 
available  at  that  time. 

In  2005,  FM  5-0  Army  Planning  and  Orders  Production 25  provided  detailed  guidance  on 
military  planning.  It  emphasized  a  planning  concept  known  as  the  Operations  Process.26  The 
Operations  Process  directs  commanders  to  understand,  visualize,  describe,  and  direct  while 
leading  and  assessing.  The  process  also  emphasized  cyclical  planning  during  which  assessment 
was  a  primary  component  throughout  to  feed  the  next  evolution  of  the  plan.27  Although  the 
distribution  of  effort  shifts  during  the  operations  process,  assessment  variables  must  be  consistent 
throughout  the  planning  effort,  (see  figure  1). 
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Figure  1:  FM  5-0:  Distribution  of  Operations  Process  Activities 

Source:  FM  5-0,  Army  Planning  and  Orders  Production  (Washington,  DC:  Government  Printing 
Office,  Jan  2005),  1-10. 


25  Field  Manual  (FM)  5-0,  Army  Planning  and  Orders  Production  (Washington,  DC: 
Government  Printing  Office,  2005),  1 . 

26  FM  5-0,  1-1. 

27  FM  5-0,  1-10. 
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Despite  the  claim  that  assessment  is  critical  to  the  operations  process,  FM  5-0  does  not 
expand  on  the  assessment  process.  The  manual  loosely  describes  what  questions  assessments 
should  be  answering,  but  provides  no  framework  for  building  an  assessment  plan.  Absent  useful 
information  regarding  assessment  in  FM  5-0,  a  military  planner  might  logically  consult  doctrinal 
manuals  for  specific  types  of  military  operations  to  uncover  a  doctrinal  framework.  In  this  case, 
the  doctrine  consulted  was  counterinsurgency  doctrine. 

In  2006,  FM  3-24  Counterinsurgency  was  the  Army’s  capstone  document  for 
understanding  and  executing  counterinsurgency  operations.  Field  Manual  3-24  was  the  Army’s 
recognition  that  counterinsurgency  warfare  differed  from  the  conventional  warfare  approach  of 
fire  and  maneuver  and  required  a  novel  approach.28  Field  Manual  3-24  sought  to  inform  military 
planners  about  insurgency  and  provide  an  approach  for  conducting  counterinsurgency 
operations.29  Although  the  original  publication  addressed  an  aspect  of  warfare  that  had  gone 
unguided  for  some  time,  its  section  on  assessments  was  deficient.30  The  manual  dedicated  a  total 
of  three  pages  and  one  table  to  the  development  and  implementation  of  measurement 
criteria.  3 'Although  the  publication  stated,  “subjective  assessment  at  all  levels  is  essential  to 
understand  the  diverse  and  complex  nature  of  COIN  operations,”  the  manual  did  not  provide  any 
additional  comments.  The  manual  provided  no  examples  of  metrics,  did  not  discuss  common 
pitfalls,  and  gave  no  instructions  for  developing  metrics.32  Furthermore,  the  table  providing  (see 
figure  2)  an  example  of  progress  indicators  presented  acts  of  violence  as  the  first  indicator, 

28  Richard  H.  Schultz  and  Andrea  J.  Dew,  “Counterinsurgency  by  the  Book,”  New  York 
Times  (August  7,  2006),  accessed  on  October  24,  2014, 

http://www.nytimes.com/2006/08/07/opinion/07shultz.html?pagewanted=all&_r=0. 

29  Field  Manual  (FM)  3-24,  Counterinsurgency  (Washington,  DC:  Government  Printing 
Office,  2006),  1. 

30  Schultz  and  Dew,  “Counterinsurgency  by  the  Book.” 

31  FM  3-24,  5-26  to  5-28. 

32  FM  3-24,  5-26. 
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followed  by  population  centric  indicators.  That  might  have  led  the  planners  to  conclude  that 
indicators  of  violence  took  precedence  over  population  centric  indicators. 


The  Army  and  the  joint  force  used  the  2006  FM  3-24  until  the  culmination  of  Operation 


Iraqi  Freedom/New  Dawn  and  applied  it  in  Afghanistan  during  Operation  Enduring  Freedom.  As 
such,  it  served  as  the  common  reference  for  counterinsurgency  operations. 


•  Acts  of  violence  (numbers  of  attacks,  fnendly/hosl-nation  casualties) 

•  Dislocated  civilians  The  number,  population,  and  demographics  of  dislocated  civilian  camps 
or  the  lack  thereof  are  a  resultant  indicator  of  overall  security  and  stability  A  drop  in  the  number 
of  people  in  the  camps  indicates  an  increasing  return  to  normalcy  People  and  families  exiled 
from  or  fleeing  their  homes  and  property  and  people  returning  to  them  are  measurable  and 
tevealiiiy  ’ 

•  Human  movement  and  religious  attendance  In  societies  where  the  culture  is  dominated  by 
religion,  activities  related  to  the  predominant  farth  may  indicate  the  ease  of  movement  and 
connoence  in  security,  peoples  use  of  free  win  and  vomion.  ana  tne  presence  of  freedom  of 
religion  Possible  indicators  include  the  following 

•  Flow  of  religious  pilgrims  or  lack  thereof 

•  Development  and  active  use  of  places  of  worship 

•  Number  of  temples  and  churches  closed  by  a  government 

•  Prp^onr#  and  activity  of  small-  anH  m#»rt  i 1  »m Wh«i  rtangpr  or  ir>v*ri.irp 
conditions  exist,  these  businesses  dose  Patrols  can  report  on  the  number  of  businesses  that 
are  open  and  how  many  customers  they  have  Tax  collections  may  indicate  the  overall  amount 
of  sales  activity 

•  Levol  of  agricultural  activity 

•  Is  a  region  or  nation  self-sustaining,  or  must  life-support  type  foodstuffs  be  imported'? 

•  How  many  acres  are  «i  cultivation?  Are  the  fields  well  maintained  and  watered"? 

•  Are  agricultural  goods  getting  to  market?  Has  the  annual  need  increased  or  decreased? 

•  Presence  or  absenco  of  associations.  The  formation  and  presence  of  multiple  political 
parties  indicates  more  involvement  ot  the  people  in  government  Meetings  of  independent 
professional  associations  demonstrate  the  viability  of  the  middle  class  and  professions.  Trade 
union  activity  indicates  worker  involvement  in  the  economy  and  politics 

•  Participation  In  elections,  especially  when  insurgents  publicly  threaten  violence  against 
participants 

•  Government  services  available  Examples  include  the  following 

-  Police  stations  operational  and  police  officers  present  throughout  the  area 

•  Clinics  and  hospitals  in  full  operation,  and  whether  new  facilities  sponsored  by  the  pnvate 
sector  are  open  and  operational 

•  Schools  and  universities  open  and  functioning 

•  Freedom  of  movement  of  people,  goods,  and  communications  This  is  a  classic  measure 
to  determine  if  on  insurgency  has  denied  orcos  in  the  physical  electronic  or  print  domains 

•  Tax  Revenue  If  people  are  paying  taxes,  this  can  be  an  indicator  of  host-nation  government 
influence  and  subsequent  ctvri  stability. 

•  Industry  exports 

•  Employmont/unomploymont  rate 

•  Availability  of  otectricity 

•  Specific  attacks  on  infrastructure 


Dislocate d  whan  a  broad  term  that  includes  a  displaced  person  an  evacuee  an  expellee  an  miernatv 

displaced  person  a  migrant  a  refugee  or  a  stateless  person  (JP  1-02)  Dislocated  ovikans  are  a  product  of  the 
de^oerate  violence  associated  with  insurgencies  and  their  counteraction  (See  FM  3-05  4CVMCRP  3-33  1A  for 
additonei  information  on  dislocated  civilians  ) 


Figure  2:  FM  3-24:  Example  Progress  Indicators 

Source:  Eleadquarters,  FM  3-24  Counterinsurgency  (Washington,  DC:  Government  Printing 
Office,  Dec  2006),  5-26  to  5-28. 

Despite  the  low  level  of  detail  found  in  both  FM  5-0  and  FM  3-24,  the  definitions  of  the 
measure  of  effectiveness  (MOE)  and  measure  of  performance  (MOP)  published  in  Joint 


Publication  1  -02  provide  planners  a  doctrinal  starting  point  for  designing  an  assessment  model. 

11 


Joint  Publication  1-02  defines  the  MOE  as  “a  criterion  used  to  assess  changes  is  systems 
behavior,  capability,  or  the  operational  environment  that  is  tied  to  measuring  the  attainment  of  an 
end  state,  achievement  of  an  objective,  or  creation  of  an  effect.”33  The  same  publication  defines 
MOP  as  “a  criterion  used  to  measure  friendly  actions  that  is  tied  to  measuring  task 
accomplishment.”34  In  nonprofessional  terms,  the  MOE  answers  the  question,  are  we  doing  the 
right  things,  while  the  MOP  asks  the  question,  are  we  doing  things  right?  Although  the  terms 
MOEs  and  MOP’s  suggest  a  very  loose  framework  to  planners  looking  to  build  an  assessment 
model,  they  do  nothing  more  than  hint  at  what  is  required  in  COIN  campaign  assessment. 

Because  the  2006  version  of  FM  3-24  provided  no  instructions  for  defining  metrics,  staff 
members  and  policy  makers  attempted  to  develop  a  methodology  through  trial  and  failure.  The 
efforts  resulted  in  the  collection  of  an  overwhelming  amount  of  data.  Without  a  tested  and  proven 
methodology,  the  data  sorted  and  scored  became  lost  in  an  anarchic  system.  The  poor  data 
management  largely  contributed  to  a  distorted  picture  of  progress.35  Upon  assuming  command  of 
ISAF,  McChrystal  established  two  separate  headquarters  to  collect  and  analyze  data,  the  ISAF 
Afghan  Assessment  Group  (AAG)  and  the  ISAF  Joint  Command  (IJC)  Information  Dominance 
Center.36  Their  purpose  was  to  create  meaningful  and  relevant  security  metrics.  A  lack  of 
complete  and  accurate  data  hampered  both  headquarters’  efforts  and  produced  a  convoluted 
picture  of  security  progress  in  Afghanistan.  Despite  major  technological  advances,  data  collection 
and  management  problems  in  Afghanistan  remained  persistent  and  slow  to  change. 


33  Joint  Publication  (JP)  1-02,  Department  of  Defense  Dictionary  of  Military  and 
Associated  Terms  (Washington,  DC:  Government  Printing  Office,  2010),  163. 

34  Ibid.,  163. 

35  Ibid.,  208-209. 

36  Connable,  Embracing  the  Fog  of  War:  Assessment  and  Metrics  in  Counterinsurgency, 

208. 
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Multiple  headquarters  in  Afghanistan  collect  and  upload  Significant  Activity  (S1GACT) 
reports  to  a  central  database  known  as  the  Combined  Information  Data  Network  Exchange 
(C1DNE).37  Senior  staff  members  then  analyze  the  C1DNE  database  to  identify  indicators  of 
progress.  Although  the  data  draws  from  multiple  sources,  it  lacks  context  and  requires  further 
scrutiny  to  determine  security  progress.  Information  uploaded  to  C1DNE  normally  provides  raw 
data  without  accompanying  narrative  or  context,  leaving  interpretation  to  the  imagination  of  the 
staff  officer.  As  such,  tracking  S1GACT  data  without  the  requisite  context  only  leads  to  the 
further  diluted  picture  of  progress. 

The  presence  or  absence  of  violence,  without  context  and  other  variables,  is  not  a 
meaningful  metric  for  determining  security  progress.  A  common  conceptual  issue  for  SIGACTs 
was  the  ability  to  distinguish  between  insurgency-related  violence  and  other  acts  of  violence,  such 
as  ethnic  or  tribal  violence.38  Without  discriminating  between  the  types  of  violence,  it  is  nearly 
impossible  to  determine  a  rise  or  fall  in  insurgent  violence  in  Afghanistan  at  any  given  time. 
Examples  of  misallocation  include  intimidation  and  violence  against  civilians,  inter  and  intra 
tribal  violence,  crime,  etc.  Additionally,  the  introduction  of  a  unit  into  a  new  area  can  also 
produce  a  significant  rise  in  S1GACT  related  violence.  As  David  Kilcullen  points  out,  “violence 
tends  to  be  high  in  contested  areas  and  low  in  government-controlled  areas.”39  This  implies  that 
the  insertion  of  an  1SAF  unit  or  ANSF  into  a  previously  unoccupied  area  will  produce  a  spike  in 
violence.  This  concept  also  applies  in  an  insurgent  controlled  area.  “But  it  is  also  low  in  enemy 
controlled  areas,  so  that  a  low  level  of  violence  indicates  that  someone  is  fully  in  control  of  a 
district,  but  does  not  tell  us  who.”40  Thus,  the  absence  of  SIGACTs  in  an  area  is  not  truly 


37  Connable,  213. 

38  Ibid.,  216. 

39  David  Kilcullen,  Counterinsurgency  (New  York:  Oxford  University  Press,  2010),  70. 

40  Ibid. 
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representative  of  security,  and  without  the  proper  analysis  and  context,  does  not  provide  a 
meaningful  metric. 

The  AAG,  one  of  the  two  headquarters  that  McChrystal  established,  relied  heavily  upon 
the  data  reported  in  C1DNE  to  produce  reports  on  security  progress.  Those  reports  then  fueled 
reports  to  strategic  and  political  leaders,  such  as  the  DOD  1230  report.  The  information  presented 
to  strategic  decision  makers  focused  solely  on  enemy  activity,  and,  thereby,  slanted  the  picture  of 
relative  success  in  Afghanistan  far  from  reality.41  According  to  a  2013  Center  for  Strategic  and 
International  Studies  (CS1S)  report,  The  Afghan  War  in  2013,  Meeting  the  Challenges  of 
Transition,  1SAF  and  the  US  reported  on  only  the  “positive”  trends  in  the  fight  against  the 
insurgents.42  The  use  of  carefully  selected  indicators  that  painted  the  picture  of  tactical  progress 
severely  tainted  the  unclassified  picture  of  the  war  in  Afghanistan  after  2009.  That  1SAF  chose  to 
highlight  the  kinetics,  in  particular,  ElAs  to  inform  their  measures  of  effectiveness,  proved  a 
deadly  practice  in  a  war  of  political  attrition.43  Further,  1SAF  failed  to  develop  metrics  that 
demonstrated  any  tactical  progress  at  the  highpoint  in  1SAF  deployments.  This  slanted  picture  led 
strategic  leaders  to  believe  1SAF  had  made  significant  progress,  which  drove  policy  decisions  on 
troop  timelines.  As  a  result,  an  insurgent  expectation  of  withdrawal  by  2014  negatively  affects 
progress  towards  stability  and  security. 

The  use  of  ElAs  from  2009-2012  as  a  reportable  metric  showed  the  most  favorable  set  of 
statistical  trends.  However,  the  metric  is  devoid  of  military  or  political  meaning.44  The  continued 
use  of  ElAs  represent  a  kinetic  focus  on  tactical  victories  in  regular  combat  that  characterized  a 


41  Cordesman,  Gold,  and  Hess,  The  Afghan  War  in  2013:  Meeting  the  Challenges  of 
Transition  Vol.  1,3. 

42  Ibid. 

43  Ibid.,  9. 

44  Cordesman,  Gold,  and  Hess,  The  Afghan  War  in  2013:  Meeting  the  Challenges  of 
Transition  Vol.  1,  14. 
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great  deal  of  US  and  1SAF  reporting  before  the  insurgency  reached  the  crisis  level  in  2008,  and  in 
ways,  strikingly  similar  to  pre-Tet  assessments  in  Vietnam.  The  E1A  metric  focuses  attention  on 
the  areas  where  US,  1SAF,  and  the  best  ANA  forces  have  a  decisive  tactical  advantage.  This 
focus  on  ElAs  ignores  the  fact  that  militant  groups  such  as  the  Taliban  are  fighting  a  political  war 
of  attrition,  and  demonstrates  that  winning  tactical  clashes  does  not  defeat  an  insurgency.  As  a 
result,  it  becomes  remarkably  difficult  to  assess  any  1SAF  success  in  light  of  the  challenges  that 
the  ANSF  face. 

Metrics  and  data  collected  in  a  COIN  environment  constitute  a  large  portion  of  a  strategic 

leader’s  decision  support  repertoire.  When  the  data  is  misused  to  support  a  public  relations 

narrative  rather  than  assess  the  relative  success  or  failure  of  a  campaign’s  approach,  the 

trustworthiness  of  data  and  the  reporting  becomes  suspect  and  is  questioned.  The  DOD  1230 

report  to  Congress,  the  Report  on  Progress  toward  Security  and  Stability  in  Afghanistan,  largely 

relies  upon  the  increase  or  decrease  in  ElAs  to  support  as  assessment  of  the  security  situation. 

The  following  observation  from  the  DOD  1230  report  released  in  December  2012  coincides  with 

completion  of  the  US  and  ISAF  troop  surge  in  Afghanistan. 

“The  improvement  in  the  security  of  the  populated  areas  of  Afghanistan  was  the  most 
significant  security-related  development  of  the  reporting  period.  Security  dramatically 
improved  in  most  of  Afghanistan’s  five  most  populous  districts,  with  ElAs  in  the  first 
nine  months  of  2012  compared  to  the  same  period  in  201 1  dropping  22  percent  in  Kabul, 
62  percent  in  Kandahar,  13  percent  in  Elerat,  88  percent  in  Mezar-e-Sharif,  and  rising  2 
percent  in  Kunduz.2  In  many  of  the  most  contested  areas  of  the  country,  insurgent  attacks 
are  becoming  more  isolated,  and  are  occurring  further  away  from  population  centers.  The 
majority  of  Afghanistan’s  405  districts  now  experience  very  low  levels  of  ElAs;  80 
percent  of  attacks  occur  in  districts  encompassing  only  20  percent  of  the  population,  and 
nearly  one  half  of  all  attacks  country-wide  occur  in  just  1 7  districts  -  that  contain  only 
five  percent  of  the  population.  The  Taliban’s  ability  to  attack  the  people  of  Afghanistan 
has  been  diminished,  particularly  in  Kandahar,  the  operational  and  ideological  base  of  the 
Taliban.”45 


45  Department  of  Defense,  DOD  1230,  Report  on  Progress  toward  Security  and  Stability 
in  Afghanistan,  3. 
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YoY  Difference 


Figure  3:  DOD  1230:  Enemy  Initiated  Attacks 

Source :  Department  of  Defense.  DOD  1230,  Report  on  Progress  Toward  Security  and  Stability  in 
Afghanistan ,  Report  to  Congress  (Washington,  DC,  December  2012),  159. 

This  quote  illustrates  the  reliance  upon  ElAs  to  determine  the  progress  of  security  within 
a  region  but  there  is  no  context  provided  that  may  allow  the  metric  to  be  of  any  value.  The  fact 
that  the  location  of  attacks  during  the  time -period  had  shifted  does  not  indicate  an  overall  increase 
in  security,  nor  does  it  provide  an  accurate  assessment  of  the  state  of  the  ongoing  insurgency.  The 
following  figure  (see  figure  3)  accompanied  the  DOD  1230  quote.  Although  it  does  provide  a 
graphical  representation  of  a  rise  and  fall  in  ElAs,  it  does  so  without  context.  There  is  no  mention 
of  troop  levels  and  their  locations,  ongoing  operations,  or  the  status  of  ANSF.  A  graphic  such  as 
this  enables  the  author  to  misrepresent  the  data  to  support  his  conclusion. 
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The  false  assessments  did  not  prompt  change  until  2013  as  annotated  in  the  July  2013 
DOD  1230  report  to  Congress  entitled  Progress  toward  Security  and  Stability  in  Afghanistan. 

“As  the  nature  of  the  conflict  has  evolved,  it  has  become  clear  that  a  tally  of  ElAs  is  not  now,  nor 
was  it  ever,  the  most  complete  measure  of  the  campaign’s  progress.”46  Although  the  transition  in 
metrics  marks  recognition  of  poor  choice,  it  does  not  replace  the  pool  of  poor  data  that  informed 
strategic  decision  makers  since  2008.  This  dilutes  the  full  picture  of  progress  of  the  campaign,  by 
conducting  assessment  from  a  mid-war  point  rather  from  the  start  and  thus  encompassing  the 
whole. 

The  COIN  environment  is  complex  and  nonlinear.  Change  in  the  environment  results 
from  the  interaction  of  multiple  factors  and  cannot  be  measured  by  a  single  performance 
parameter.  1SAF  has  continued  use  of  ElAs  as  a  performance  measure  departs  from  complexity 
theory  and  does  not  provide  useful  information  to  strategic  and  political  leaders.  This  fact, 
however,  does  not  prove  that  military  leaders  intentionally  deceived  strategic  decision-makers  by 
their  reports  but  does  show,  nonetheless,  negligence  in  providing  strategic  leaders  information 
within  the  correct  context.  The  improved  approach  towards  assessment,  touted  in  the  2013  DOD 
1230  report,  has  yet  to  produce  evidence  of  its  success.  Elosts  of  assessment  methods,  such  as  the 
Measuring  Progress  in  Conflict  Environments  Metrics  (MP1CE)  Framework  or  District  Stability 
Framework  (DSF),  have  recently  received  attention  in  various  professional  journals,  but  lack 
proven  effectiveness.  Regardless  of  the  updated  model,  since  1945  counterinsurgencies  have 
taken  a  minimum  of  fourteen  years  to  succeed.  Various  collection  methodologies  and  metrics 


46  Department  of  Defense,  1230  Report,  Progress  toward  Stability  and  Security  in 
Afghanistan,  2013,  8. 
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present  at  best  a  snapshot  of  a  phase  rather  than  the  campaign  in  its  whole.  Longitudinal  data  is 
unavailable  and  the  causes  of  long-term  effects  remain  unidentified.47 

Had  ISAF  integrated  additional  factors  to  support  their  performance  measures  a  clearer 
picture  of  progress  might  have  been  available.  The  following  section  presents  a  different 
approach  toward  security  metrics  employed  towards  the  end  of  the  American  experience  in 
Vietnam,  and  with  it,  a  different  way  to  think  about  a  complex  problem. 

The  American  Experience  in  Vietnam 

The  Vietnam  War  presents  a  useful  case  study  of  assessment  measures  because  the 
United  States  faced  in  Vietnam  the  same  difficulties  developing  useful  assessments  as  in 
Afghanistan.  In  both  conflicts,  strategic  decision-makers  had  to  determine  the  progress  of  stability 
and  security  despite  inaccurate  information  that  was  decontextualized  and  did  not  provide  a  clear 
understanding  of  the  environment.48  The  Vietnam  campaign  assessment  model  was  arguably  the 
most  complex  and  detailed  model  that  history  has  to  offer.  Although  the  US  initial  efforts  to 
conduct  campaign  assessment  in  Vietnam  were  unsuccessful,  changes  in  the  assessment  system 
occurred  much  faster  than  in  Afghanistan.  The  changes  in  assessment  methods  evolved  into  an 
assessment  model  known  as  the  Hamlet  Evaluation  System  (HES).  Introduced  in  1967,  the 
Military  Assistance  Command  Vietnam  (MACV)  used  the  HES  throughout  the  remainder  of  US 
operations  in  Vietnam.49 


47  Seth  G.  Jones,  Counterinsurgency  in  Afghanistan,  RAND  Counterinsurgency  Study, 
Volume  4  (Santa  Monica:  RAND  Publications,  2008),  5.  This  RAND  study  claims  that  successful 
counterinsurgencies  since  1945  require  a  minimum  of  fourteen  years  commitment,  while 
unsuccessful  ones  average  eleven  years. 

48  Connable,  Embracing  the  Fog  of  War:  Assessment  and  Metrics  in  Counterinsurgency, 

140. 

49  Stathis  N.  Kaylvas  and  Matthew  Kocher,  Violence  and  Control  in  Civil  War:  An 
Analysis  of  the  Hamlet  Evaluation  System  (Chicago:  University  of  Chicago,  2009),  3. 
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The  origins  of  the  US  decision  to  employ  COIN  tactics  in  South  Vietnam  are  traceable  to 
a  visit  by  Sir  Robert  G.K.  Thompson  to  South  Vietnam’s  president  Ngo  Dinh  Diem.50  During  the 
visit,  Thompson  convinced  Diem  to  employ  a  combined  military  and  civil  operations  approach  to 
clear  Viet-Cong  controlled  areas  in  South  Vietnam.  That  advice  resounded  with  General  Maxwell 
Taylor,  then  Chairman  of  Joint  Chiefs  of  Staff,  and  helped  convince  President  John  F.  Kennedy  to 
commit  troops  to  Vietnam.51  The  result  was  the  establishment  of  MACV  in  February  1962,  and 
the  embarkation  on  a  strategy  to  isolate  South  Vietnamese  citizens  from  the  communist  Viet 
Cong.  The  methodology  adopted  to  do  so,  known  as  the  strategic  hamlet  program,  moved 
villagers  into  defendable  locations  in  which  the  South  Vietnamese  government  could  provide 
adequate  security.52  The  approach  of  separating  the  population  from  the  Viet-Cong  insurgents 
was  one  embraced  by  Kennedy  as  evidenced  by  his  speech  to  the  West  Point  graduating  class  of 
1962.  “If  freedom  is  to  be  saved,  we  need  a  whole  new  kind  of  strategy,  a  wholly  different  kind  of 
force,  and  a  wholly  different  kind  of  training  and  commitment.’53  Despite  Kennedy’  enthusiasm, 
he  did  not  communicate  a  concept  for  combating  Communist-based  insurgency.  As  was  the  case 
during  the  early  years  in  Afghanistan,  counterinsurgency  doctrine  was  vague  and  agencies 
supporting  the  effort  coordinated  poorly.  54  Those  facts,  however,  did  not  delay  the  commitment 
of  troops  as  advisors.  By  the  summer  of  1963,  16,500  military  advisors  were  in  South  Vietnam.55 


50  Graham  A.  Cosmos,  MACV:  The  Joint  Command  in  the  Years  of  Escalation,  1962- 
1967  (Washington,  DC:  Center  of  Military  History,  2006),  76. 

51  Cosmos,  76-77. 

52  Ibid.,  77-78. 

53  Herbert  R.  McMaster,  Dereliction  of  Duty  (New  York:  HarperCollins  Publishers, 
1997),  32. 

54  Ibid. 

55  Ibid.,  32-33. 


19 


Following  the  overthrow  and  assassination  of  Diem,  and  the  subsequent  assassination  of 


Kennedy,  the  public  and  the  Congress  began  to  scrutinize  US  efforts  in  Vietnam.  In  late 
December  1963,  Secretary  of  Defense  Robert  S.  McNamara  returned  from  a  visit  to  Vietnam  with 
the  belief  that  if  current  trends  not  reversed  in  the  next  two  to  three  months,  South  Vietnam  would 
more  than  likely  become  a  communist-controlled  state.56  This  assessment,  along  with 
recommendations  made  by  Taylor,  led  President  Lyndon  Johnson  to  issue  his  first  major  Vietnam 
policy  directive.  NSAM  288,  issued  on  March  17  1964,  reiterated  US  commitment  to  help  South 
Vietnam  defeat  the  Viet-Cong  and  set  into  motion  actions  that  would  lead  to  “graduated  overt 
military  pressures”  against  North  Vietnam.57  A  major  aspect  of  the  graduated  response  was  the  air 
campaign,  known  as  Operation  Rolling  Thunder,  which  targeted  Viet-Cong  military  and  supply 
targets  in  South  Vietnam.  Johnson  had  not  yet  made  the  decision  to  commit  additional  ground 
troops,  but  remained  committed  to  “taking  any  action  necessary  to  preserve  South  Vietnam.”58 
Following  a  visit  to  South  Vietnam,  Johnson  held  a  meeting  with  the  Joint  Chiefs  of  Staff  on 
March  15,  1965  to  charge  them  with  finding  new  ways  to  improve  the  combat  situation.  During 
the  meeting,  he  stated  that  he  wanted  greater  use  of  air  power  and  high  numbers  of  dead  Viet- 
Cong.  He  concluded  the  meeting  by  stating  the  he  wanted  a  weekly  report  on  the  number  of  Viet- 
Cong  killed.59  As  such,  MACV  began  structuring  itself  to  provide  that  detail  in  their  reports,  and 
Viet-Cong  body  count  became  the  accepted  measure  to  assess  success  in  South  Vietnam.  This 
approach  towards  campaign  assessment  continued  for  nearly  four  years. 


56  Cosmos,  MACV:  The  Joint  Command  in  the  Years  of  Escalation,  1962-1967,  117. 

57  Ibid.,  119-120. 

58  McMaster,  Dereliction  of  Duty,  248-249. 

59  Ibid. 
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The  use  of  body  counts  as  a  performance  measure  in  Vietnam  greatly  inflated  the 
perception  of  progress  and  shaped  military  unit  and  leader  behavior,  which  eventually  led  to  a 
devastating  surprise  when  the  North  Vietnamese  Army  (NVA)  launched  the  Tet  Offensive  in 
1968.  Given  the  complexity  of  the  COIN  environment,  performance  measures  built  on  a  single 
variable  do  not  add  to  an  understanding  of  the  environment,  but  rather  contribute  to  entropy  in  a 
failing  system.60  Given  a  lack  of  guiding  doctrine  at  the  time,  and  the  desire  to  show  military 
progress  to  strategic  leaders  in  Washington,  MACV  provided  statistics  on  enemy  casualties  that 
were  often  overinflated  and  that  painted  an  unrealistic  picture  of  progress.61  The  reports  to 
strategic  leaders  were  so  bloated  that  by  1968,  the  figures  of  enemy  casualties  approached  the 
estimated  total  strength  of  the  entire  number  of  Viet-Cong  as  well  as  the  full  Tet  attack  force.62 
Junior  intelligence  officers  in  the  MACV  J-2  made  formal  complaints  that  senior  commanders 
were  misusing  data  and  understating  enemy  numbers  while  overstating  enemy  losses.  This  further 
fueled  an  intense  debate  between  the  Central  Intelligence  Agency  (CIA)  and  MACV  over  the 
original  order  of  battle  numbers  for  both  the  Viet-Cong  and  the  NVA.  That  debate  eventually  led 
to  complete  recounting  of  enemy  forces.63  The  danger  of  changing  a  defined  variable,  such  as 
initial  enemy  strength,  halfway  through  a  campaign  is  that  it  hinders  time-series  analysis  and 
complicates  the  assessment  of  progress.  It  may  even  reduce  the  value  of  accurate  reporting. 

Following  the  Tet  Offensive  in  1968,  the  picture  that  MACV  had  painted  for  strategic 
leaders  in  Washington  had  nearly  no  value  for  estimating  campaign  progress.  This  led  to  growing 
dissent  among  the  American  public,  and  frustration  and  distrust  within  the  Johnson 

60  Axelrod  and  Cohen,  Harnessing  Complexity:  Organizational  Implications  of  a 
Scientific  Frontier,  119-121. 

61  Cosmos,  MACV:  The  Joint  Command  in  the  Years  of  Withdrawal,  1968-1973,  86-87. 

62  Ibid. 

63  Cosmos,  MACV:  The  Joint  Command  in  the  Years  of  Withdrawal,  1968-1973,  87. 
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administration.  The  overly  optimistic  MACV  reporting — demanded  by  political  leaders — created 
a  tremendous  divide  between  military  and  strategic  political  leaders  and  clouded  the  future  of  US 
involvement  in  Vietnam.64 

The  US  and  1SAF  initial  efforts  to  assess  progress  in  Afghanistan  closely  mirrors  the 
efforts  of  MACV  in  Vietnam.  At  the  start  of  their  respective  efforts,  both  headquarters  lacked  a 
guiding  doctrine  to  assist  in  the  development  of  assessment  criteria  in  the  COIN  environment  and 
both  developed  an  approach  over  time  through  trial  and  error.  The  early  use  of  a  single  measure 
to  inform  strategic  leaders  was  present  in  both  assessment  models,  leading  to  an  uncertain  picture 
of  progress.  The  use  of  ELAs  by  1SAF  shows  only  the  impact  of  violence  in  those  areas  that  have 
some  degree  of  troop  presence.  As  Kilcullen  earlier  identified,  the  level  of  violence  within 
contested  areas  tends  to  be  higher  than  government-controlled  areas.65  This  dynamic  renders  the 
E1A  as  a  single  indicator  of  progress  useless  for  informing  strategic  political  leadership. 

Similarly,  the  use  of  body  counts  as  a  single  indicator  of  progress  by  MACV  met  Johnson’s 
druthers  but  proved  valueless  to  strategic  leaders.  Unlike  Afghanistan  many  years  later,  Johnson 
compelled  MACV  to  produce  reports  based  on  body  counts.66  Furthermore,  the  shock  brought  on 
by  the  Tet  Offensive,  an  operation  thought  by  some  to  be  impossible  because  of  the  inflated 
reporting,  drove  a  significant  gap  between  military  and  civilian  leaders.  The  use  of  body  counts  in 
Vietnam  serves  as  a  cautionary  tale  for  any  planner.  The  application  of  attrition  warfare  measures 
to  COIN  not  only  further  convoluted  an  already  complex  environment,  but  also  had  unintended 
consequences  at  the  tactical  level.67  The  appeal  of  reporting  higher  body  counts  to  represent 


64  Ibid.,  104. 

65  Kilcullen,  Counterinsurgency,  70. 

66  Connable,  Embracing  the  Fog  of  War:  Assessment  and  Metrics  in  Counterinsurgency, 


151-152. 


67  Ibid.,  153. 
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greater  effectiveness  in  combat  occurred  often  at  the  tactical  level,  ultimately  resulting  in 
confusion  of  progress  at  the  tactical  level  as  well  as  the  strategic. 

A  Change  in  Approach 

Strategic  leaders  in  Washington  eventually  recognized  a  need  for  a  change  in  strategy  in 
Vietnam.  The  change  occurred  in  1968  when  General  Creighton  Abrams’  assumed  command  of 
MACV.68  The  eventual  change  included  a  restructured  approach  towards  measuring  campaign 
progress  known  as  the  Hamlet  Evaluation  System  (HES).  Although  imperfect,  the  HES  measured 
a  larger  spectrum  of  variables  than  in  Afghanistan.  As  such,  a  reasonable  degree  of  accuracy  was 
determinable  by  a  recognized  rise  and  falls  in  variables.  The  following  section  will  review  the 
reaction  by  strategic  political  leaders  and  military  commanders  to  the  initial  measures  used  in 
Afghanistan  and  Vietnam.  This  section  will  also  analyze  the  changes  in  doctrine  and  campaign 
assessment  models  by  both  1SAF  and  MACV  in  their  respective  theaters.  At  conclusion,  this 
section  will  highlight  those  aspects  of  change,  which  are  of  value  to  campaign  assessment  in  the 
COIN  environment. 

Afghanistan  and  1SAF 

In  May  2014,  the  US  Army  published  a  new  version  of  FM  3-24  Insurgencies  and 
Countering  Insurgencies.  The  manual  improves  upon  the  2006  version.69  It  dedicates  a  chapter  to 
campaign  assessment  that  includes  sections  on  assessment  frameworks,  methods,  and 
considerations.  The  manual  also  expands  the  original  sections  on  the  development  of 
measurement  criteria.  The  previous  version  of  FM  3-24  did  not  provide  this  level  of  detail,  which 
indicates  that  lessons  neglected  in  Iraq  and  Afghanistan. 


68  Fewis  Sorley,  Thunderbolt  From  the  Battle  of  the  Bulge  to  Vietnam  and  Beyond: 
General  Creighton  Abrams  and  the  Army  of  His  Times  (New  York,  NY :  Simon  and  Schuster, 
1992),  255. 

69  FM  3-24,  1. 
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There  are  several  themes  that  permeate  each  section  of  the  assessments  chapter,  largely 


revolving  around  the  importance  of  drawing  from  several  variables  in  the  environment  and 
viewing  them  in  aggregate.  The  doctrine  further  provides  assessment  considerations,  emphasizing 
the  fact  that  a  counterinsurgency  environment  is  a  political  problem,  and  that  assessments  should 
focus  on  understanding  and  solving  the  political  problem.  The  section  titled  “Afghanistan  and 
1SAF”  cautions  against  reliance  upon  any  assessment  method  that  claims  to  provide  information 
on  the  underlying  social,  cultural,  political,  and  economic  turbulence  rapidly. 70  The  section 
dedicated  to  designing  measures  of  effectiveness  and  performance  notes  that  the  raw  count  of 
incidents  provides  little  useful  information,  particularly,  when  compared  to  assessments  of  the 
scale  or  sophistication  of  an  attack. 7 'Finally,  the  publication  states  that  above  all  else,  the  context 
in  which  a  measure  is  used  is  extremely  important.  The  fact  that  schools  are  being  built  does  not 
mean  more  students  are  attending  school,  and,  thus,  addressing  a  political  need  to  provide 
schooling.  The  construction  of  schools  is  a  measure  of  performance.  Unfortunately,  it  is  not  also  a 
measure  of  effectiveness.  The  section  also  cautions  against  the  use  of  enemy  casualties  as  a 
measure  of  progress,  since  greater  casualties  may  indicate  an  increase  in  recruits  and  enemy 
aggressiveness.72  The  revised  FM  3-24  emphasized  the  importance  of  context  and  a  focus  beyond 
a  single  measure  in  the  assessment  process.  This  effort  does  well  to  describe  the  necessary 
attributes  of  a  COIN  assessment  model.  Flowever,  the  does  not  explain  how  to  define  an 
assessment  model  nor  it  provide  detailed  example  models. 

The  revised  FM  3-24  does  provide  some  detail  concerning  the  development  of  measures 
and  certain  guiding  principles,  but  leaves  room  for  improvement.  The  major  failure  of  the  later 


70  FM  3-24,  12-3. 

71  Ibid.,  12-4. 

72  Ibid.,  12-5. 
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version  is  its  lack  of  example  assessment  frameworks  for  staffs  and  commanders  to  build  upon. 
The  doctrine  does  refer  to  one  example,  the  Interagency  Conflict  Assessment  Framework,  but 
does  not  provide  context  or  detail  for  execution.73  As  such,  the  Army  and  the  joint  force  are  to 
develop  a  framework  through  further  trial  and  error,  partially  defeating  the  purpose  of  a  revised 
doctrine. 

In  addition  to  the  revised  doctrine,  there  have  been  several  publications  released  by 
research  organizations  and  various  other  groups  to  assist  in  the  development  of  a  COIN  campaign 
assessment  model.  To  date,  doctrine  integrated  none  of  these  studies.74  The  2015  National 
Security >  Strategy >  (NSS)  emphasizes  transferring  responsibility  to  ANSF  and  the  Government  of 
the  Islamic  Republic  of  Afghanistan  (GIRoA).75  Consequently,  the  DOD  1230  reports  now 
provide  a  focus  on  transition  to  Afghan  authorities  (See  figure  4)  rather  than  a  focus  on  security 
and  stability,  as  the  title  of  the  document  touts.  The  focus  on  transition  did  not  bring  with  it  a 
refined  approach  to  assessing  security  and  stability.  Thus,  the  shift  in  emphasis  toward  transition 
diverts  attention  from  those  aspects  of  the  environment  that  prove  to  be  indicators  of  progression 
or  regression. 


73  Ibid.,  12-1. 

74  The  Measuring  Progress  in  Conflict  Environments  (MP1CE)  framework,  published  by 
the  United  States  Institute  of  Peace  in  2010,  is  the  most  detailed  example. 

75  Barack  Obama,  National  Security  Strategy >  2015  (Washington,  DC,  Government 
Printing  Office,  2015),  9-10. 
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Annex  A:  Security  Metrics  and  Assessments20 

ANA  Assessment  and  RDL  Matrix  (CSS) 


Figure  4:  DOD  1230:  Security  Metrics  Assessments 

Source:  Department  of  Defense.  DOD  1230,  Report  on  Progress  Toward  Security  and  Stability  in 
Afghanistan.  Report  to  Congress,  (Washington  DC.  October  2014),  A-l. 


Despite  the  claim  of  an  improved  approach  to  assessing  progress  in  security  and  stability 
in  Afghanistan,  there  is  no  evidence  that  the  quality  of  reports  have  improved.  The  July  2013 
DOD  1230  report,  which  identified  the  overreliance  upon  a  single  measure,  touted  renewed 
attention  to  economic  development  and  improvements  in  ANSF  capacity.76  This  change  in  focus 
however  does  not  improve  ISAF’s  ability  to  measure  security  and  stability.  Instead,  it  only 
changes  the  data  ISAF  chooses  to  collect,  correlate,  and  report.  There  is  no  evidence  of  a  change 
in  collection  methodology  to  integrate  the  appropriate  environmental  context,  and  there  is  no 
evidence  to  support  the  claim  of  increased  emphasis  on  economic  growth  and  development.  In 


76  Department  of  Defense,  DOD  1230,  Report  on  Progress  toward  Security  and  Stability 
in  Afghanistan  (July  2013),  8-10. 
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fact,  as  of  January  2015, 1SAF  classified  the  data  dealing  with  the  expenditure  of  funds  allocated 
to  training  the  ANSF  as  well  as  data  on  reconstruction  and  infrastructure  improvement  efforts. 
This  data  was  previously  releasable.77  The  classification  of  previously  unclassified  data  indicates 
distrust  in  the  ability  for  ISAF  to  assess  progress,  and  an  internal  lack  of  confidence  in  the  system 
built  to  do  so. 

Although  many  are  clearly  dissatisfied  with  ISAFs  approach  to  campaign  assessment,  the 
changes  made,  primarily  in  Army  doctrine,  were  not  enough  to  improve  the  quality  of  reports  to 
strategic  political  leaders.  Reporting  that  is  not  holistic,  and  presents  a  single  measure  as  an 
indicator  of  progress  does  not  assist  strategic  leaders  with  the  information  for  sound  decision¬ 
making,  the  primary  purpose  of  such  reports  as  the  DOD  1230.  The  record  shows  that  the  US  and 
ISAF  did  not  draw  appropriate  historical  lessons  from  the  experience  in  Vietnam.  Similar  to 
ISAF,  MACV  in  Vietnam  had  an  opportunity  to  assess  its  methodology  and  make  necessary 
adjustments.  Although  the  HES  did  not  reach  its  highest  level  of  sophistication  until  nearly  the 
US  withdrawal  from  Vietnam,  it  does  provide  a  useful  model  from  which  contemporary  staffs  can 
build  an  effective  campaign  assessment  model.  The  following  section  will  analyze  said  changes 
and  indicate  which  ones  are  particularly  useful  today. 

Vietnam  and  MACV 

Beginning  in  1 967,  MACV  faced  a  steadily  growing  requirement  for  collection  and 
reporting.  McNamara  drove  the  requirements  and  they  nearly  overwhelmed  the  MACV  staff.78 
To  relieve  the  stress  placed  on  MACV,  Johnson  made  the  decision  in  1967  to  establish  a 

77  Phillipa  H.  Stewart,  “Quietly  Classifying  US  Spending  in  Afghanistan,”  AlJazeera, 

War  and  Conflict  (February  3,  2015),  accessed  February  1 1,  2015, 

http://www.aliazeera.com/indepth/features/2015/02/quietly-classifying-spending-afghanistan- 
150201 141449193.html 

78  Cosmos,  MACV:  The  Joint  Command  in  the  Years  of  Escalation,  1962-1967,  290. 
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supporting  headquarters  known  the  Office  of  Civil  Operations  and  Revolutionary  Development 
Support  (CORDS).  The  president  gave  CORDS  the  responsibility  for  pacification  in  South 
Vietnam  and  appointed  Ambassador  Robert  Komer,  a  former  Army  officer,  as  the  director. 
Komer  was  the  only  civilian  to  possess  control  over  military  operations  in  Vietnam.79  Under 
Komer,  the  HES  framework  for  assessment  began  implementation.  The  system  collected  and 
correlated  reports  that  fueled  analysis  in  the  Office  of  Systems  Analysis  back  in  Washington.80 
The  CIA  originally  created  the  HES  for  use  by  CORDS  in  response  to  dissatisfaction  with  the 
original  “body  count”  reporting.  It  existed  as  part  of  the  Pacification  Evaluation  System 
(PACES),  a  fully  automated  system  designed  to  determine  who  controlled  the  population  of 
South  Vietnam.81 

Although  HES  was  once  controversial,  recent  researchers  have  found  the  HES  to  be 
useful  for  understanding  the  complexities  of  assessment  in  the  COIN  environment.  The  HES 
provides  a  starting  point  for  a  holistic  assessment  framework.  According  to  Stathis  Kalyvas,  “The 
HES  is  likely  to  remain  for  some  time  the  only  systematic  micro-level  database  of  civil  war 
dynamics  covering  such  a  large  territory  and  time  period.”82  In  his  analysis,  Kalyvas  addresses 
criticisms  of  the  data,  but  still  finds  the  HES  to  be  a  “unique  resource  for  the  study  of  the 
dynamics  of  civil  wars  in  terms  of  its  scope,  detail,  and  level  of  analysis.”83  A  1975  Naval  War 
College  study  of  the  HES  found  that  despite  a  degree  of  bias  induced  by  District  Senior  Advisors 


79  Cosmos,  MACV:  The  Joint  Command  in  the  Years  of  Escalation,  1962-1967,  360-362. 

80  Connable,  Embracing  the  Fog  of  War:  Assessment  and  Metrics  in  Counterinsurgency, 

138. 

81  Douglas  Kinnard,  The  War  Managers  (Annapolis:  Naval  Institute  Press,  1977),  107. 

82  Stathis  N.  Kalyvas  and  Matthew  K ocher,  “The  Dynamics  of  Violence  in  Vietnam:  An 
Analysis  of  the  Hamlet  Evaluation  System  (HES)”  Journal  of  Peace  Research,  Vol.  46,  No  3 
(May  2009),  342-343. 

83  Ibid.,  343. 
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(DSAs)  the  measurement  of  the  HES  pacification  indicators  was  quite  satisfactory  .  .  .  and  that 
the  HES  [ought  to]  be  retained  for  further  development,  and  that  a  proponent  [ought  to]  be 
designated.”84  Additionally,  a  RAND  study  of  the  statistical  item  analysis  of  the  December  1967 
HES  data  by  Anders  Sweetland  found  the  system  to  be  meaningful  and  mathematically  stable.85 
Given  the  amount  of  scholarly  research  that  found  value  in  the  HES,  there  is  value  in  analyzing 
the  framework  to  determine  what  attributes  of  the  framework  are  applicable  to  an  assessment 
framework  for  Afghanistan. 

The  HES  geographical  framework  appears  remarkably  similar  to  what  1SAF  is  attempting 
to  achieve  in  its  revised  campaign  plan.  Each  province,  district,  village,  and  hamlet  in  South 
Vietnam  possessed  a  unique  HES  identification  number  and  designated  spatial  coordinates.  With 
the  cooperation  of  local  officials,  US  District  Senior  Advisors  (DSAs)  evaluated  the  status  of 
each  hamlet  and  village  under  their  jurisdiction  on  a  monthly  and  quarterly  basis.  The  HES  was 
originally  an  attempt  to  systemize  and  digitize  a  less  formal  prior  practice  of  the  South 
Vietnamese  government  to  assign  letter  ratings  to  the  security  status  at  the  local  level.86  The  HES 
data  collection  occurred  via  a  series  of  worksheets  provided  at  the  district  and  provincial  level  by 
provincial  advisory  teams.87  The  worksheets  themselves  resembled  the  modem  bubble  sheet. 


84  Albert  C.  Bole,  An  Evaluation  of  the  Measurements  of  the  Hamlet  Evaluation  System 
(Newport:  Center  for  Advanced  Research,  Naval  War  College  Press,  1975),  ii. 

85  Anders  Sweetland,  Item  Analysis  of  the  HES  (Santa  Monica:  RAND  Coiporation, 
1968),  ii. 

86  Kalyvas  and  Kocher,  Violence  and  Control  in  Civil  War:  An  Analysis  of  the  Hamlet 
Evaluation  System,  3. 

87  Connable,  Embracing  the  Fog  of  War:  Assessment  and  Metrics  in  Counterinsurgency, 

154. 
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The  HES  questionnaire  rated  six  major  “factors,”  equivalent  to  the  modem  day 
MOEs/MOPs,  and  had  spaces  for  narrative  inputs.88  It  required  DSAs  to  rate  Hamlets  from  A-E 
or  V  on  18  indicators  of  relative  insurgent/govemment  control.  Averaged  numerical  responses  to 
these  questions  provided  the  baseline  data  for  reports  forward  to  the  OSA  in  Washington.89 


Status 

Frequency 

Percent 

Cumulative 

Percent 

A 

581 

10.52 

10.52 

B 

2035 

36.85 

47.37 

C 

802 

14.52 

61.89 

D 

212 

3.84 

65.73 

E 

144 

2.61 

68.33 

V 

11 

0.20 

68.53 

Figure  5:  HES  Worksheet  Ratings 

Source :  Stathis  N.  Kalyvas  and  Matthew  Kocher,  Violence  and  Control  in  Civil  War:  An  Analysis 
of  the  Hamlet  Evaluation  System  (Chicago:  University  of  Chicago,  2007),  10. 

The  HES  underwent  a  series  of  changes  over  a  six-year  period.  The  rapid  changes  to  the 
HES  in  Vietnam  eventually  produced  in  a  useful  framework.  However,  when  viewed  in  aggregate 
there  is  a  distorted  picture  of  progress.  A  major  source  of  controversy  over  the  earlier  versions  of 
the  HES  (1967-1969)  stemmed  from  the  influence  military  commanders  had  over  responses  to  the 
HES  questionnaires.  The  later  versions  of  HES  (1970-1971)  had  respondents  fill  out  more 
qualitative  questionnaires,  with  analysts  at  a  level  above  the  local  commander  level  assigning  the 
codes,  or  values  to  the  data.90  At  the  cost  of  time -series  comparisons,  the  frequent  debates  over 
the  HES  definitions  occurred,  and  changes  to  the  pre-coded  ratings  occurred  nearly  annually 


88  Ibid. 

89  Kalyvas  and  Kocher,  Violence  and  Control  in  Civil  War:  An  Analysis  of  the  Hamlet 
Evaluation  System,  9. 

90  Lewis  Sorley,  Vietnam  Chronicles:  the  Abrams  Tapes,  1968-1972  (Lubbock:  Texas 
Tech  University  Press,  2004),  331. 
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through  1973. 91  A  change  in  a  fixed  variable  promoted  further  debate  within  MACV,  CORDS, 
and  the  OSA.  Another  initial  problem  with  the  HES  was  how  data  averaging  occurred  to 
represent  progress.  Because  Hamlets  received  a  score  based  on  several  indicators  on  a  scale  of  1- 
5,  analysts  used  an  average  to  provide  a  single  overall  rating.92  A  former  HES  manager,  Colonel 
Erwin  R.  Brigham  explained  how  this  process  worked.  “A  hamlet  with  a  security  sector  rating  of 
3  As,  4  Bs,  and  2Cs;  and  a  development  sector  rating  of  2  Cs,  4  Ds,  and  3  Es  would  receive  an 
overall  rating  of  C.  Similarly,  A,  B,  D,  and  E  hamlets  may  have  individual  ratings  of  A,  B,  C,  D, 
and  E.”93  The  problem  with  this  approach  was  that  by  providing  an  average,  it  did  not  help  the 
analyst  to  understand  the  context  of  the  environment.  Was  development  inconsequential  due  to  a 
lack  of  security?  How  did  the  average  villager  view  his  or  her  own  security?  Without  appropriate 
context,  the  qualitative  aspects  of  the  assessed  hamlets  remained  hidden  and,  thus,  left  to  the 
report  author’s  imagination. 

The  averaged  data  fueled  reports  typically  delivered  to  senior  decision  makers.  As  such, 
the  results  of  such  reports  were  an  average  based  upon  a  series  averages  to  inform  policy. 
Averages  present  a  danger  to  accurate  assessment.  Averages  smooth  out  large  differences,  which 
reduced  the  transparency  and  accuracy  of  the  HES  data  to  a  point,  that  the  overall  number  was 
nearly  meaningless.94  Furthermore,  strategic  political  leadership  used  the  HES  to  present  progress 
in  pacification,  completely  out  of  context.  George  Allen,  one  of  the  most  respected  intelligence 


91  Connable,  Embracing  the  Fog  of  War:  Assessment  and  Metrics  in  Counterinsurgency, 
157. 

92  Connable,  157. 

93  Erwin  R.  Brigham,  “ Pacification  Measurement  in  Vietnam:  The  Hamlet  Evaluation 
System Civil  Operations  for  Revolutionary  Development  Support  (Vietnam:  Headquarters, 
Military  Assistance  Command,  1968),  9-10. 

94  Connable,  Embracing  the  Fog  of  War:  Assessment  and  Metrics  in  Counterinsurgency, 

166. 
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analysts  during  the  Vietnam  provided  an  account  of  how  the  HES  results  were  misunderstood  and 
misused: 

[W]e  got  pressure  from  the  White  House,  the  way  they  used  the  Hamlet  Evaluation 
System.  They  were  the  ones  that  decided  this  A-B-C-D-E,  or  5-4-3-2-1  scale,  you  could 
come  up  with  an  aggregate  average  score  for  all  the  hamlets  in  Vietnam,  monthly,  and  it 
was  2.163  maybe  this  month.  And  they  would  sit  there  with  [bated]  breath  waiting  to  see 
what  next  month’s  score  would  be.  And  if  it  were  2.165,  two  one-thousandths  higher  than 
last  month,  “Great,  we’re  progressing,  we’re  progressing!95 

Allen’s  insight  provides  a  lens  into  how  the  HES  and  its  associated  reports  influenced 
strategic  policy  makers.  Congressman  John  Tunney,  a  sitting  member  of  Congress  at  the  time, 
shared  Allen’s  perspective.  According  to  Tunney,  “Presidential  advisors  have  briefed  Members  of 
the  Congress  and  told  them  that  a  certain  percentage  usually  carried  to  one  or  two  decimal  places, 
of  the  South  Vietnamese  people  were  living  in  relatively  secure  areas  under  the  control  of  the 
Saigon  government.”  He  further  asserted  “this  information  was  being  used  at  the  highest  levels  of 
the  US  government  as  a  guide  to  operational  planning  and  as  a  means  of  informing  the  American 
people  of  the  progress  of  the  war.”96  What  the  HES  lacked  was  the  context  to  accompany  the 
data.  The  lack  of  balance  between  qualitative  and  quantitative  data,  contextualized  beyond  the 
efforts  of  an  intelligence  analyst,  left  the  consumers  of  the  HES — strategic  policy  makers — 
dissatisfied  with  the  information  they  were  receiving,  and  suspicious  of  the  military  headquarters 
providing  the  reports.  Given  the  poor  ratings  of  the  HES,  this  begs  the  question  of  what  lessons 
could  be  drawn  from  the  HES  and  applied  to  contemporary  campaign  assessment?  Thomas 
Thayer,  author  of  War  without  Fronts  and  former  director  of  the  Southeast  Asia  Intelligence  and 
Force  Effectiveness  Division  (SA1FED)  provides  some  clues  to  an  answer. 


95  Mai  Elliott,  RAND  in  Southeast  Asia:  A  History  of  the  Vietnam  War  Era  (Santa 
Monica:  RAND  Corporation,  2010),  93. 

96  John  V.  Tunney,  Measuring  Hamlet  Security  in  Vietnam:  Report  of  a  Special  Study 
Mission  (Washington,  DC:  US  Government  Printing  Office,  December  1968),  1. 


32 


Thayer  initially  served  as  a  lower  level  analyst  during  the  Vietnam  War  before  rising  to 
be  the  SA1FED  director  within  the  OS  A/'7  His  efforts  while  serving  in  that  capacity  provided  the 
best  comprehensive  campaign  assessment  in  COIN.  Thayer  published  the  Southeast  Asia  Analysis 
Report,  later  republished  in  a  twelve-volume  series  titled  A  Systems  Analysis  View  of  the  Vietnam 
War.  Although  his  reports  were  unofficial,  due  to  their  being  the  only  effort  to  aggregate  the  war 
reporting,  they  became  widely  read  and  influential  within  Washington98  He  later  published  War 
without  Fronts,  which  was  a  critique  of  his  previous  efforts  to  analyze  the  assessment  efforts. 
Although  he  was  critical  of  much  of  the  averaged  data,  he  presented  the  concept  of  “reasonable 
accuracy,”  which  finds  value  in  the  trends  presented  by  the  HES.  Thayer  believed  that  despite  the 
inaccuracy  of  field  reporting,  a  reasonable  degree  of  accuracy  to  show  useful  patterns  and  trends 
is  possible.99  This  concept  presents  an  interesting  proposal  but  requires  a  baseline  aggregate  from 
which  to  determine  trends.  What  Thayer  lacked  at  the  time  and  remains  undeveloped,  is  a  useful 
doctrine  that  expands  upon  his  reasonable  accuracy  concept. 

MACV’s  experience  with  the  application  of  the  HES  provides  a  useful  case  study  from 
which  to  develop  a  doctrinal  framework  for  campaign  assessment  in  COIN.  The  transition  from  a 
focus  on  a  single  measure,  to  other  aspects  of  progress,  represents  an  ability  to  derive  lessons 
from  negative  feedback.  The  integration  of  the  HES  framework  however  never  achieved  its  goal 
of  informing  strategic  decision  makers.  This  failure  resulted  from  presenting  the  aggregate 
assessment  based  on  averages  separated  from  the  geographic  context.  The  fact  however  remains 
that  the  trial  and  failure  effort  of  the  HES  provides  an  excellent  historical  example  from  which  to 
build  an  accurate  COIN  campaign  assessment  framework. 

97  Connable,  Embracing  the  Fog  of  War:  Assessment  and  Metrics  in  Counterinsurgency, 

141. 

98  Connable,  157. 

99  Thomas  Thayer,  War  without  Fronts:  The  American  Experience  in  Vietnam  (Boulder: 
Westview  Press,  1985),  152. 
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Comparing  the  Frameworks 

The  Vietnam-era  experience  with  assessments  provides  valuable  historical  lessons.  The 
need  for  a  balance  between  quantitative  and  qualitative  measures,  Thayer’s  concept  of  reasonable 
accuracy,  and  the  importance  of  keeping  the  variables  consistent  are  a  few  of  the  invaluable 
lessons  learned  by  MACV  over  a  six-year  period.  This  experience  could  provide  1SAF  a  pool  of 
knowledge  from  which  to  develop  potential  frameworks  for  assessment.  To  determine  whether 
1SAF  has  improved  its  measures,  a  comparison  of  the  1SAF  framework  to  the  lessons  provided  by 
the  Vietnam-era  experience  is  necessary. 

The  July  2013  DOD  1230  report  touted  a  change  in  approach  towards  assessment,  having 
recognized  an  overreliance  upon  violence  as  an  indicator  of  progress. 100  Since  then  the  US  and 
1SAF  have  claimed  to  use  a  series  of  frameworks  to  feed  assessments,  derived  from  a  series  of 
factors  beyond  violence.  At  present,  it  is  unclear  what  assessment  framework  ISAF  has  adopted. 
District  Stability  Framework  (DSF),  a  US  Agency  for  International  Development  (USAID) 
model,  has  received  much  attention  in  professional  journals,  and  ISAF  has  acknowledged  that  it  is 
used.  According  to  ISAF,  “The  DSF  is  designed  to  directly  target  the  root  causes  of  instability 
using  carefully  thought  out,  cost  effective  activities.”101  Along  with  the  identification  of  causes  of 
instability,  the  DSF  claims  to  establish  a  baseline  to  measure  effects  over  time.102 


100  Department  of  Defense,  DOD  1230,  Report  on  Progress  toward  Security  and  Stability 
in  Afghanistan  (July  2013),  8. 

101  NATO.  “District  Stability  Framework  in  Afghanistan.”  last  modified  March  13,  2012, 
accessed  February  10,  2015,  http://www.aco.nato.int/district-stability-framework-in- 
afghanistan.aspx. 

102  Mirko  L.  Cmkovich,  “District  Stability  Framework:  Social  Science  Underpinnings  of 
Complex  Operations”  (presented  at  the  MORS  Mini-Symposium  through  the  US  Agency  for 
International  Development,  Office  of  Military  Affairs)  accessed  October  22,  2014, 

http  ://www.  dtic.mil/ dtic/tr/ fulltext/u2/ a53  5679.pdf 
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The  Counterinsurgency  Training  Center-Afghanistan  (CTC-A)  and  USAID  developed  the 
DSF  through  a  cooperative  effort  in  2012.  Per  the  published  DSF  “quick  reference  guide,”  the 
purpose  of  DSF  is  to  “utilize  a  range  of  situational  awareness  and  planning  tools  to  inform 
stability  planning.”103  The  DSF  claims  to  develop  a  baseline  for  measuring  whether  stability  is 
increasing  or  decreasing  through  use  of  a  field-level  tool  focused  on  variables  beyond  violence 
(see  figure  6). 

USAID  District  Stability  Framework  (DSF) 

fROM  THt  AMERICAN  P£ORU 


1.  Focuses  on  Stabilization  not  Development 

2.  Provides  a  common  civil-military  operating  picture 

3.  Is  a  field-level  analysis,  planning,  and  programming 
tool 

4.  Provides  information  for  integration  into  higher  level 
civilian-military  planning 

5.  Creates  a  baseline  to  measure  whether  stability  is 
increasing/decreasing 


Figure  6:  DSF  Focus  Areas 

Source :  Mirko  L.  Cmkovich,  “District  Stability  Framework:  Social  Science  Underpinnings  of 
Complex  Operations”  (presented  at  the  MORS  Mini-Symposium  through  the  US  Agency  for 
International  Development,  Office  of  Military  Affairs)  accessed  October  22,  2014, 
http://www.dtic.mil/dtic/tr/fulltext/u2/a535679.pdf. 


The  DSF  appears  to  be  an  improvement  from  the  SIGACT  collection  effort  by  the  AAG. 
Through  its  field  tools  and  inquiry  process,  the  DSF  claims  to  be  able  to  measure  variables 
ranging  from  government  legitimacy  to  security  perceptions  (see  Figure  7).  The  expansion  of 


103  US  Agency  for  International  Development,  District  Stability  Framework  Quick 
Reference  Guide,  accessed  February  22,  2014,  1, 

http://www.mccdc.marines.mil/Portals/172/Docs/SWCIWID/STAB/DSF_Quick_Reference_Gui 
de_Dec_20 1 0-COP-239.pdf. 
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variables  theoretically  should  inform  stability  and  security  better  than  single  indicators  of 
violence. 


USAID 


Monitoring  &  Evaluation 


Three  Levels  of  Evaluation: 

1 .  Were  the  activities  completed?  (MOP/Output) 

2.  Were  the  intended  effects  achieved?  (MO E/Impact) 

3.  Is  Overall  Stability  increasing?  For  example: 

I.  Government  Legitimacy 

II.  Local  on  Local  Violence 

III.  Economic  Health 

IV.  Local  Security  Forces  Presence 

V.  Freedom  of  Movement 

VI.  Governance  Perceptions 

VII.  Security  perceptions 

KEY:  DSF  establishes  a  baseline  to  measure  effects  over  time 

15 


Figure  7:  DSF  Measures 

Source :  Mirko  L.  Cmkovich,  “District  Stability  Framework:  Social  Science  Underpinnings  of 
Complex  Operations”  (presented  at  the  MORS  Mini-Symposium  through  the  US  Agency  for 
International  Development,  Office  of  Military  Affairs)  accessed  October  22,  2014, 
http://www.dtic.mil/dtic/tr/fulltext/u2/a535679.pdf. 


In  order  to  achieve  the  three  levels  of  evaluation  referenced  in  Figure  7,  the  DSF  claims 
to  employ  a  cyclical  approach  that  include  waypoints  of  situational  awareness,  analysis,  design, 
and  monitoring  and  evaluation  respectively. 104  To  achieve  a  degree  of  situational  awareness  that 
informs  progress,  the  DSF  claims  to  use  a  four-step  process  (see  figure  8)  of  environmental  factor 
analysis  to  build  situational  awareness.  In  addition  to  the  cyclical  framework,  DSF  provides  a 
series  of  matrices,  generic  questionnaires,  and  other  synchronization  tools  to  assist  in  collection  to 


104  Mirko  L.  Cmkovich,  “District  Stability  Framework:  Social  Science  Underpinnings  of 
Complex  Operations,”  accessed  October  22,  2014, 
http://www.dtic.mil/ dtic/tr/ fulltext/u2/ a53  5679.pdf. 
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inform  assessment.  At  a  surface  level,  this  approach  takes  into  account  many  of  the  variables  that 
the  HES  attempted  to  address  the  pacification  efforts  in  South  Vietnam. 


USAID 

THE  AMERICAN  PEOPLE 


Situational  Awareness 


1.  Identify  Stability  and  Instability  factor $  and  their  relevance  to  the  population  using  the  Operational 
Variables  of  PMESII  and  the  civil  considerations  of  ASCOPE  for  a  local  Operating  Environment. 


2.  Understand  the  Cultural  Environment  by  determining  cultural  specific  dynamics  that  exist  in  the  local 
environment;  traditional  conflict  resolution  mechanisms,  traditional  authorities,  limits  to  the  traditional 
mechanisms  and  how  Anti-Government  Elements  exploit  them. 


3.  Apply  Stability  and  Instability  Dynamics  using  gained  situational  awareness  to  further  refine  possible 
factors  of  Instability,  possible  factors  of  Stability  (Resiliencies),  and  determine  what  Key  Actors  with  the 
means  and  motivations  to  foster  stability  and  instability  are  present  in  a  local  area. 


4.  Understand  Perception  Dynamics  by  determining  priority  issues  (grievances)  that  are  effecting  the 
local  population.  Identify  gaps  in  knowledge  about  a  local  operating  environment  and  confirm  or  deny 
our  assumptions  using  multiple  (available)  sources  and  types  of  perception  data. 


Figure  8:  DSF  Situational  Awareness  Approach 

Source :  Mirko  F.  Cmkovich,  “District  Stability  Framework:  Social  Science  Underpinnings  of 
Complex  Operations”  (presented  at  the  MORS  Mini-Symposium  through  the  US  Agency  for 
International  Development,  Office  of  Military  Affairs)  accessed  October  22,  2014, 
http://www.dtic.mil/dtic/tr/fulltext/u2/a535679.pdf. 


The  DSF  provides  a  compilation  of  assessment  tools  and  guidance  for  the  development  of 
stability  and  security.  This  framework  does  appear  to  draw  from  the  variables  addressed  by  the 
HES,  particularly  attitudinal,  security,  and  governance  perceptions.  What  the  DSF  lacks,  what 
1SAF  does  not  address,  and  what  the  HES  was  most  deficient  in,  is  the  methodology  or  form  of 
collection  for  reasonable  accuracy. 

One  of  the  major  criticisms  of  the  HES  was  the  corruption  of  data  by  field  commanders 
with  competing  interests.  A  major  source  of  controversy  in  the  earlier  versions  of  the  HES  (1967- 
1969)  was  the  local  military  commander’s  influence  over  responses  to  the  HES  questionnaires. 
The  later  versions  of  HES  (1970-1971)  had  respondents  fill  out  more  qualitative  questionnaires, 
with  analysts  at  a  level  above  the  local  commander  level  assigning  the  codes,  or  values  to  the 
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data.105  There  is  no  evidence  that  the  US  and  ISAF  have  restructured  its  approach  to  reach  the 
level  of  sophistication  acquired  through  trial  and  error  in  Vietnam.  This  means  that  ISAF  must 
not  only  adopt  a  framework  which  uses  measures  beyond  acts  of  violence,  but  must  also  address 
the  form  and  function  of  collection  to  improve  assessment  accuracy.  This  requires  a  reliance  upon 
data  beyond  SIGACTs  and  reports  compiled  at  the  battalion  and  brigade  level.  To  reach  the  level 
of  sophistication  that  the  FIES  had,  a  dedicated  effort  is  required  avoid  the  natural  tendency 
amongst  commanders  to  adjust  data  to  show  greater  progress.  The  value  of  the  HES  was  not 
simply  the  variables  measured,  but  rather  the  system  in  its  whole,  how  it  approached  measures 
and  sought  to  improve  itself.  Although  ISAF  has  recognized  a  deficiency  in  the  measures 
collected  and  the  quality  of  its  assessments,  it  has  not  made  the  necessary  adjustments  to  produce 
assessments  that  are  useful  for  strategic  and  political  leader  decision  making. 

Conclusion 

Senior  political  leadership  and  military  professionals  have  expressed  dissatisfaction  with 
assessments  produced  to  show  the  progress  toward  stability  and  security  in  Afghanistan.  This 
sentiment  derives  from  the  fact  that  the  US  and  fSAF  measured  progress  in  stability  and  security 
solely  through  a  rise  and  fall  in  acts  of  violence — in  particular,  enemy  initiated  ambushes. 106  A 
campaign  requires  accurate  assessments,  derived  from  carefully  selected  measures  to  inform 
decision-making.  Despite  recognizing  the  flaw,  there  is  no  evidence  that  ISAF  has  yet  developed 
a  more  effective  framework.  The  lack  of  progress  is  attributable  to  a  dearth  of  useful  doctrine  and 
the  inability  of  ISAF  to  develop  an  approach  informed  by  historical  examples.  Despite  the  revised 


105  Sorley,  Vietnam  Chronicles:  The  Abrams  Tapes,  1968-1972,  331. 

106  Cordesman,  Gold,  and  Fless,  The  Afghan  War  in  2013:  Meeting  the  Challenges  of 
Transition  W ol.  1,  10. 
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doctrine,  the  US  and  ISAF  might  have  accelerated  insight  through  the  review  of  variables 
developed  by  the  HES  and  the  lessons  learned  by  MACV  in  its  assessment  framework. 

In  order  to  determine  progress  over  time,  an  assessment  plan  should  be  in  place  prior  to 
execution  of  a  campaign.  This  affords  strategic  leaders  and  operational  commanders  alike  the 
ability  to  view  longitudinal  data  and  determine  causality  of  long-term  effects. 107  Without  a  useful 
assessment  plan,  success  and  failure  is  easily  misattributed,  leading  to  a  misconstrued  strategy. 
This  fact  requires  military  planners  to  dedicate  effort  to  the  development  of  an  assessment  plan 
with  measures  related  to  the  context  of  the  campaign  and  its  desired  end  state.  The  current 
doctrine  loosely  describes  what  assessment  plans  should  consist  of,  but  fails  to  provide  a  useful 
framework,  context,  or  historical  lessons  learned.  A  revision  of  the  current  doctrine  is  required  to 
rectify  this  problem.  Such  a  revision  requires  an  example  framework  that  does  not  just  describe  a 
framework,  but  explain  the  complexity  of  accurate  assessment  in  COIN,  provide  a  useful 
framework  and  example  measures,  recommend  collection  methodologies  and  task  organization, 
as  well  as  provide  historical  examples. 

Despite  a  lack  of  guiding  doctrine,  the  HES  and  MACV  experiences  in  the  development 
of  assessment  plans  provide  a  useful  historical  example  from  which  ISAF  can  development  an 
accurate  assessment  plan.  The  HES,  although  initially  flawed,  continuously  improved  to  meet  the 
requirements  of  senior  strategic  leaders.  The  change  in  variables  caused  an  inability  to  view 
longitudinal  date,  similar  to  the  current  situation  in  Afghanistan.  Regardless,  the  steps  taken  by 
MACV  to  prevent  corruption  of  data  by  field  commanders  and  improve  the  quality  of  reporting 
provide  insight  into  the  complexity  of  campaign  assessment  in  COIN,  and  valuable  lessons  for 
military  planners. 


107  Reilly,  Operational  Design:  Distilling  Clarity  from  Complexity  for  Decisive  Action, 

114-115. 
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The  military  planner  draws  approaches  and  plans  not  only  from  doctrine,  but  also  from 
theory  and  history.  Given  a  lack  of  useful  doctrine,  the  military  planner  should  look  to  other 
sources  to  inform  his  efforts.  The  US  and  ISAF’s  assessment  plan  failed  to  reach  the  level  of 
sophistication  achieved  by  MACV  in  Vietnam.  Although  1SAF  recognized  fault  in  its  assessment 
plan,  and  claims  to  have  changed  its  approach,  there  is  no  evidence  to  support  such  a  claim.  The 
integration  of  DSF  as  an  assessment  model  provides  a  useful  example  for  what  assessment  is 
supposed  to  achieve,  but  does  not  address  the  challenges  of  data  collection  nor  what  a 
headquarters  should  do  with  such  data.  The  evolution  of  the  FIES  provides  the  necessary  context 
in  which  a  military  planner  can  build  an  accurate  assessment  plan.  The  failure  of  the  US  and 
1SAF  to  cull  the  appropriate  lessons  from  history  has  led  to  uninformed  assessments  and 
dissatisfied  strategic  and  political  leaders.  As  such,  military  planners  cannot  afford  to  ignore  the 
haunting  rhymes  of  history,  and  the  lessons  provided  by  the  TIES  and  MACV. 
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